{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 使用 LLM 作为评判者🧑‍⚖️进行自动化和多方面的评估\n",
    "_作者: [Aymeric Roucher](https://huggingface.co/m-ric)_\n",
    "\n",
    "评估大型语言模型（LLMs）通常是一项困难的任务：由于他们能力广泛，给它们分配的任务时通常应该根据非常泛且松散的要求来判断。例如，AI 对问题的回答可能是：\n",
    "- 不基于上下文\n",
    "- 重复、重复、重复\n",
    "- 语法错误\n",
    "- 过于冗长，用词过多，导致话语或书面内容过于详细和拖沓\n",
    "- 不连贯\n",
    "- ...\n",
    "\n",
    "这些标准的列表还有很多。即使我们有一个有限的列表，每一个标准的衡量都是困难的：\"制定一个基于规则的程序来评估输出是非常具有挑战性的。传统的评估指标，基于输出和参考答案之间的相似性（例如，ROUGE、BLEU），对于这些问题也无效。\"\n",
    "\n",
    "✅ 一种强大的解决方案，可以在不需要昂贵人力的前提下，以人类的方式评估输出，就是使用 LLM 作为评判者。\n",
    "这种方法在 [《Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena》](https://huggingface.co/papers/2306.05685) 中被介绍 - 推荐阅读这篇文章。\n",
    "\n",
    "💡 这个想法很简单：让 LLM 为你评分。 🤖✓ \n",
    "但我们将会看到，它不能直接很好地适配：你需要仔细设置才能得到好的结果。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install huggingface_hub datasets pandas tqdm -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "import pandas as pd\n",
    "from tqdm.auto import tqdm\n",
    "from datasets import load_dataset\n",
    "from huggingface_hub import InferenceClient, notebook_login\n",
    "\n",
    "tqdm.pandas()  # load tqdm's pandas support\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "\n",
    "notebook_login()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\nI’m good, thanks. I’m in the middle of a tour at the'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "repo_id = \"mistralai/Mixtral-8x7B-Instruct-v0.1\"\n",
    "\n",
    "llm_client = InferenceClient(\n",
    "    model=repo_id,\n",
    "    timeout=120,\n",
    ")\n",
    "\n",
    "# Test your LLM client\n",
    "llm_client.text_generation(prompt=\"How are you today?\", max_new_tokens=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 准备创建和评估我们的 LLM 评判者\n",
    "\n",
    "假设你想给 LLM 一个特定任务，比如回答开放式问题。\n",
    "\n",
    "困难在于，正如我们上面讨论的，衡量答案质量是困难的，例如，精确的字符串匹配会错误地将许多正确但措辞不同的答案标记为错误。\n",
    "\n",
    "你可以让人类标签员评判输出，但这会花费他们很多时间，如果你想更新模型或问题，你必须重新做一遍。\n",
    "\n",
    "✅ 在这种情况下，你可以设置一个 LLM 作为评判者。\n",
    "\n",
    "**但是要使用 LLM 作为评判者，你首先需要评估它对模型输出的评分有多可靠。**\n",
    "\n",
    "➡️ 所以第一步将是... 创建一个人工评估数据集。但你只需要为少数示例获取人工标注 - 大约 30 个应该足以对性能有一个好的了解。\n",
    "\n",
    "每次你想测试你的 LLM 作为评判者时，你都可以重新使用这个数据集。\n",
    "\n",
    "在我们的案例中，我们将使用 [`feedbackQA`](https://huggingface.co/datasets/McGill-NLP/feedbackQA)，它包含每个问题/答案对的 2 个人类评估和评分：使用 30 个示例的样本将代表你的小型评估数据集可能的样子。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ratings = load_dataset(\"McGill-NLP/feedbackQA\")[\"train\"]\n",
    "ratings = pd.DataFrame(ratings)\n",
    "\n",
    "ratings[\"review_1\"] = ratings[\"feedback\"].apply(lambda x: x[\"rating\"][0])\n",
    "ratings[\"explanation_1\"] = ratings[\"feedback\"].apply(lambda x: x[\"explanation\"][0])\n",
    "ratings[\"review_2\"] = ratings[\"feedback\"].apply(lambda x: x[\"rating\"][1])\n",
    "ratings[\"explanation_2\"] = ratings[\"feedback\"].apply(lambda x: x[\"explanation\"][1])\n",
    "ratings = ratings.drop(columns=[\"feedback\"])\n",
    "\n",
    "# Map scores to numeric values\n",
    "conversion_dict = {\"Excellent\": 4, \"Acceptable\": 3, \"Could be Improved\": 2, \"Bad\": 1}\n",
    "ratings[\"score_1\"] = ratings[\"review_1\"].map(conversion_dict)\n",
    "ratings[\"score_2\"] = ratings[\"review_2\"].map(conversion_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算性能基准线是一个好主意：例如，这里可以是两个人类评分者之间的评分一致性，通过他们给出的分数的[皮尔逊相关系数](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient)来衡量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation between 2 human raters:\n",
      "0.563\n"
     ]
    }
   ],
   "source": [
    "print(\"Correlation between 2 human raters:\")\n",
    "print(f\"{ratings['score_1'].corr(ratings['score_2'], method='pearson'):.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "两个真人评委之间的相关性并不是那么好。如果你们的真人评分真的很差，这可能意味着评分标准不够清晰。\n",
    "\n",
    "这意味着我们的“真实情况”包含了一些噪音：因此我们不能期望任何算法评估能够非常接近它。\n",
    "\n",
    "然而，我们可以减少这种噪音：\n",
    "- 通过取平均分作为我们的真实情况，而不是任何一个单独的分数，我们应该能够平衡一些不规则性。\n",
    "- 只选择人类评审员达成一致意见的样本。\n",
    "\n",
    "在这里，我们将选择最后一个选项，并且**只保留两个人类评审员达成一致意见的示例**。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>review_1</th>\n",
       "      <th>explanation_1</th>\n",
       "      <th>review_2</th>\n",
       "      <th>explanation_2</th>\n",
       "      <th>score_1</th>\n",
       "      <th>score_2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>human_score</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What can I do to help people that are grieving?</td>\n",
       "      <td>Coping with Stress\\nTake care of yourself and your community\\nTaking care of yourself, your friends, and your family can help you cope with\\nstress. Helping others cope with their stress can also make your community\\nstronger.\\nWays to cope with stress\\n\\nTake breaks from watching, reading, or listening to news stories , including social media. Hearing about the pandemic repeatedly can be upsetting.\\nTake care of your body. \\nTake deep breaths, stretch, or meditate.\\nTry to eat healthy, well-balanced meals.\\nExercise regularly, get plenty of sleep.\\nAvoid alcohol and drugs.\\n\\n\\nMake time to unwind. Try to do some other activities you enjoy.\\nConnect with others. Talk with people you trust about your concerns and how you are feeling.\\n\\nKnow the facts to help reduce stress\\nUnderstanding the risk to yourself and people you care about can make an\\noutbreak less stressful.\\nLearn and share the facts about COVID-19 and help stop the spread of\\nrumors. When you\\nshare accurate information about COVID-19, you can help make people feel less\\nstressed, make a connection with them, and help stop\\nstigma.\\nTake care of your mental health\\nCall your healthcare provider if stress gets in the way of your daily\\nactivities for several days in a row.\\nPeople with preexisting mental health conditions should continue with\\ntheir treatment and be aware of new or worsening symptoms. Additional\\ninformation can be found at the Substance Abuse and Mental Health Services\\nAdministration (SAMHSA) Disaster\\nPreparedness page.\\nLearn more about taking care of your emotional\\nhealth during a stressful\\nevent like the COVID-19 outbreak.</td>\n",
       "      <td>Bad</td>\n",
       "      <td>The question is about others which the reply did not answer.</td>\n",
       "      <td>Bad</td>\n",
       "      <td>The response could have addressed how to help those that are grieving cope rather than what it was presenting.</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What protocols do workplaces need to follow to keep everyone safer?</td>\n",
       "      <td>Coronavirus and Australian workplace laws\\nHealth &amp; safety in the workplace\\nWorkplaces must follow the rules about health and safety during coronavirus to\\nhelp stop it spreading. Find out more about:\\n\\nrules and obligations under workplace health and safety laws\\nhow to manage the risk of coronavirus in the workplace\\nwhere to go for help.\\n\\nLearn more about Health and safety in the workplace during\\ncoronavirus.</td>\n",
       "      <td>Could be Improved</td>\n",
       "      <td>This answer needs to be improved because it doesn’t provide information up-front about workplaces during the pandemic. Instead, it just includes a hyperlink.</td>\n",
       "      <td>Could be Improved</td>\n",
       "      <td>there is one link to information, but there is no information in the answer about how to stay safe in the workplace. it talks about the need to stay safe in the workplace, but it doesn't talk about ways in which to actually do that.</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How soon can I apply for financial support?</td>\n",
       "      <td>COVID-19 early release of super\\nAfter you apply\\nIt will take us up to four business days to process your application and send\\nyour outcome letter to your myGov inbox. You may also receive an SMS\\nnotification.\\nIf you receive a notification from us and haven't applied to access your super\\nearly, you need to call us or your fund as soon as possible.\\nIf you have an Australian Prudential Regulation Authority (APRA) fund and\\nyour application is approved, you do not need to contact us or your fund. Your\\nfund will make the payment to you without you needing to apply to them\\ndirectly.\\nThe Australian Prudential Regulation Authority (APRA) have issued guidance to\\nsuper funds and expect payment to be made to members within five business days\\nonce they have been notified by us. However, this time may increase where\\nfunds need to contact you to clarify information. More information can be\\nfound on APRA's websiteExternal Link.\\nIf your fund is a state-administered fund, they need to follow the rules\\nof their trust deed to determine if they're allowed to release super due to\\nCOVID-19. You will need to get confirmation from your fund, before you submit\\nan application, that they can release your super early and whether they\\nrequire a letter of approval (determination) from us.\\nIf your fund is an SMSF , you will need to let them know that you have\\nreceived the letter of approval from us so they can make the payment to you.</td>\n",
       "      <td>Acceptable</td>\n",
       "      <td>There is information on how to apply for the help.  Still, there is nothing say how long you have to wait before applying.</td>\n",
       "      <td>Acceptable</td>\n",
       "      <td>This response says how long the applications take to process and then some more information about the process. There's a link to more relevant information. A pretty good answer</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Should vulnerable children be expected to be in educational settings?</td>\n",
       "      <td>Guidance Actions for schools during the coronavirus outbreak\\nPrioritising pupils\\nWhat are our expectations regarding vulnerable children and young people attending educational settings?\\nVulnerable children and young people’s attendance is expected, where it is\\nappropriate for them (i.e. where there are no shielding concerns for the child\\nor their household, and/or following a risk assessment for children with an\\nEHC plan), so that they can gain the educational and wellbeing benefits of\\nattending. Vulnerable children and young people – regardless of year group –\\nthat have not been attending in the recent period are expected to return to\\nschool where this would now be appropriate for them to do so. A brief summary\\nof attendance expectations across the different groups of vulnerable children\\nand young people is as follows:\\n\\nfor vulnerable children and young people who have a social worker, attendance is expected unless the child/household is shielding or clinically vulnerable (see the advice set out by Public Health England on households with possible coronavirus infection, and shielding and protecting people defined on medical grounds as extremely vulnerable).\\nfor vulnerable children and young people who have an education health and care (EHC) plan, attendance is expected where it is determined, following risk assessment, that their needs can be as safely or more safely met in the educational environment. Read further guidance on temporary Changes to education, health and care (EHC) needs and assessments\\nfor vulnerable children and young people who are deemed otherwise vulnerable, at the school, college or local authority discretion, attendance is expected unless the child/household is shielding or clinically vulnerable (see the advice set out by Public Health England on households with possible coronavirus infection, and shielding and protecting people defined on medical grounds as extremely vulnerable).\\n\\n*[EHC]: Education, Health and Care</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>There is a lot of relevant information here.  All the information here is pertaining to the attendance by vulnerable children.</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>This answers the questions and includes links and guides on how to help keep the kids healthy. It provides guidelines on what to do and how to bring the students back to school</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                          question  \\\n",
       "human_score                                                                          \n",
       "1                                  What can I do to help people that are grieving?   \n",
       "2              What protocols do workplaces need to follow to keep everyone safer?   \n",
       "3                                      How soon can I apply for financial support?   \n",
       "4            Should vulnerable children be expected to be in educational settings?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            answer  \\\n",
       "human_score                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          \n",
       "1                                                                                                                                                                                                                                                                                                                                                                           Coping with Stress\\nTake care of yourself and your community\\nTaking care of yourself, your friends, and your family can help you cope with\\nstress. Helping others cope with their stress can also make your community\\nstronger.\\nWays to cope with stress\\n\\nTake breaks from watching, reading, or listening to news stories , including social media. Hearing about the pandemic repeatedly can be upsetting.\\nTake care of your body. \\nTake deep breaths, stretch, or meditate.\\nTry to eat healthy, well-balanced meals.\\nExercise regularly, get plenty of sleep.\\nAvoid alcohol and drugs.\\n\\n\\nMake time to unwind. Try to do some other activities you enjoy.\\nConnect with others. Talk with people you trust about your concerns and how you are feeling.\\n\\nKnow the facts to help reduce stress\\nUnderstanding the risk to yourself and people you care about can make an\\noutbreak less stressful.\\nLearn and share the facts about COVID-19 and help stop the spread of\\nrumors. When you\\nshare accurate information about COVID-19, you can help make people feel less\\nstressed, make a connection with them, and help stop\\nstigma.\\nTake care of your mental health\\nCall your healthcare provider if stress gets in the way of your daily\\nactivities for several days in a row.\\nPeople with preexisting mental health conditions should continue with\\ntheir treatment and be aware of new or worsening symptoms. Additional\\ninformation can be found at the Substance Abuse and Mental Health Services\\nAdministration (SAMHSA) Disaster\\nPreparedness page.\\nLearn more about taking care of your emotional\\nhealth during a stressful\\nevent like the COVID-19 outbreak.   \n",
       "2                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             Coronavirus and Australian workplace laws\\nHealth & safety in the workplace\\nWorkplaces must follow the rules about health and safety during coronavirus to\\nhelp stop it spreading. Find out more about:\\n\\nrules and obligations under workplace health and safety laws\\nhow to manage the risk of coronavirus in the workplace\\nwhere to go for help.\\n\\nLearn more about Health and safety in the workplace during\\ncoronavirus.   \n",
       "3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         COVID-19 early release of super\\nAfter you apply\\nIt will take us up to four business days to process your application and send\\nyour outcome letter to your myGov inbox. You may also receive an SMS\\nnotification.\\nIf you receive a notification from us and haven't applied to access your super\\nearly, you need to call us or your fund as soon as possible.\\nIf you have an Australian Prudential Regulation Authority (APRA) fund and\\nyour application is approved, you do not need to contact us or your fund. Your\\nfund will make the payment to you without you needing to apply to them\\ndirectly.\\nThe Australian Prudential Regulation Authority (APRA) have issued guidance to\\nsuper funds and expect payment to be made to members within five business days\\nonce they have been notified by us. However, this time may increase where\\nfunds need to contact you to clarify information. More information can be\\nfound on APRA's websiteExternal Link.\\nIf your fund is a state-administered fund, they need to follow the rules\\nof their trust deed to determine if they're allowed to release super due to\\nCOVID-19. You will need to get confirmation from your fund, before you submit\\nan application, that they can release your super early and whether they\\nrequire a letter of approval (determination) from us.\\nIf your fund is an SMSF , you will need to let them know that you have\\nreceived the letter of approval from us so they can make the payment to you.   \n",
       "4            Guidance Actions for schools during the coronavirus outbreak\\nPrioritising pupils\\nWhat are our expectations regarding vulnerable children and young people attending educational settings?\\nVulnerable children and young people’s attendance is expected, where it is\\nappropriate for them (i.e. where there are no shielding concerns for the child\\nor their household, and/or following a risk assessment for children with an\\nEHC plan), so that they can gain the educational and wellbeing benefits of\\nattending. Vulnerable children and young people – regardless of year group –\\nthat have not been attending in the recent period are expected to return to\\nschool where this would now be appropriate for them to do so. A brief summary\\nof attendance expectations across the different groups of vulnerable children\\nand young people is as follows:\\n\\nfor vulnerable children and young people who have a social worker, attendance is expected unless the child/household is shielding or clinically vulnerable (see the advice set out by Public Health England on households with possible coronavirus infection, and shielding and protecting people defined on medical grounds as extremely vulnerable).\\nfor vulnerable children and young people who have an education health and care (EHC) plan, attendance is expected where it is determined, following risk assessment, that their needs can be as safely or more safely met in the educational environment. Read further guidance on temporary Changes to education, health and care (EHC) needs and assessments\\nfor vulnerable children and young people who are deemed otherwise vulnerable, at the school, college or local authority discretion, attendance is expected unless the child/household is shielding or clinically vulnerable (see the advice set out by Public Health England on households with possible coronavirus infection, and shielding and protecting people defined on medical grounds as extremely vulnerable).\\n\\n*[EHC]: Education, Health and Care   \n",
       "\n",
       "                      review_1  \\\n",
       "human_score                      \n",
       "1                          Bad   \n",
       "2            Could be Improved   \n",
       "3                   Acceptable   \n",
       "4                    Excellent   \n",
       "\n",
       "                                                                                                                                                             explanation_1  \\\n",
       "human_score                                                                                                                                                                  \n",
       "1                                                                                                             The question is about others which the reply did not answer.   \n",
       "2            This answer needs to be improved because it doesn’t provide information up-front about workplaces during the pandemic. Instead, it just includes a hyperlink.   \n",
       "3                                               There is information on how to apply for the help.  Still, there is nothing say how long you have to wait before applying.   \n",
       "4                                           There is a lot of relevant information here.  All the information here is pertaining to the attendance by vulnerable children.   \n",
       "\n",
       "                      review_2  \\\n",
       "human_score                      \n",
       "1                          Bad   \n",
       "2            Could be Improved   \n",
       "3                   Acceptable   \n",
       "4                    Excellent   \n",
       "\n",
       "                                                                                                                                                                                                                                        explanation_2  \\\n",
       "human_score                                                                                                                                                                                                                                             \n",
       "1                                                                                                                                      The response could have addressed how to help those that are grieving cope rather than what it was presenting.   \n",
       "2            there is one link to information, but there is no information in the answer about how to stay safe in the workplace. it talks about the need to stay safe in the workplace, but it doesn't talk about ways in which to actually do that.   \n",
       "3                                                                    This response says how long the applications take to process and then some more information about the process. There's a link to more relevant information. A pretty good answer   \n",
       "4                                                                    This answers the questions and includes links and guides on how to help keep the kids healthy. It provides guidelines on what to do and how to bring the students back to school   \n",
       "\n",
       "             score_1  score_2  \n",
       "human_score                    \n",
       "1                  1        1  \n",
       "2                  2        2  \n",
       "3                  3        3  \n",
       "4                  4        4  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Sample examples\n",
    "ratings_where_raters_agree = ratings.loc[ratings[\"score_1\"] == ratings[\"score_2\"]]\n",
    "examples = ratings_where_raters_agree.groupby(\"score_1\").sample(7, random_state=1214)\n",
    "examples[\"human_score\"] = examples[\"score_1\"]\n",
    "\n",
    "# Visualize 1 sample for each score\n",
    "display(examples.groupby(\"human_score\").first())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 创建我们的 LLM 评判者\n",
    "\n",
    "我们使用一个基本提示来构建我们的 LLM 评判者，包含以下元素：\n",
    "- 任务描述\n",
    "- 标度描述：`最小值`，`最大值`，值类型（这里为`浮点数`）\n",
    "- 输出格式的解释\n",
    "- 一个答案的开头，尽可能引导 LLM\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "JUDGE_PROMPT = \"\"\"\n",
    "You will be given a user_question and system_answer couple.\n",
    "Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.\n",
    "Give your answer as a float on a scale of 0 to 10, where 0 means that the system_answer is not helpful at all, and 10 means that the answer completely and helpfully addresses the question.\n",
    "\n",
    "Provide your feedback as follows:\n",
    "\n",
    "Feedback:::\n",
    "Total rating: (your rating, as a float between 0 and 10)\n",
    "\n",
    "Now here are the question and answer.\n",
    "\n",
    "Question: {question}\n",
    "Answer: {answer}\n",
    "\n",
    "Feedback:::\n",
    "Total rating: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples[\"llm_judge\"] = examples.progress_apply(\n",
    "    lambda x: llm_client.text_generation(\n",
    "        prompt=JUDGE_PROMPT.format(question=x[\"question\"], answer=x[\"answer\"]),\n",
    "        max_new_tokens=1000,\n",
    "    ),\n",
    "    axis=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_judge_score(answer: str, split_str: str = \"Total rating:\") -> int:\n",
    "    try:\n",
    "        if split_str in answer:\n",
    "            rating = answer.split(split_str)[1]\n",
    "        else:\n",
    "            rating = answer\n",
    "        digit_groups = [el.strip() for el in re.findall(r\"\\d+(?:\\.\\d+)?\", rating)]\n",
    "        return float(digit_groups[0])\n",
    "    except Exception as e:\n",
    "        print(e)\n",
    "        return None\n",
    "\n",
    "\n",
    "examples[\"llm_judge_score\"] = examples[\"llm_judge\"].apply(extract_judge_score)\n",
    "# Rescale the score given by the LLM on the same scale as the human score\n",
    "examples[\"llm_judge_score\"] = (examples[\"llm_judge_score\"] / 10) + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation between LLM-as-a-judge and the human raters:\n",
      "0.567\n"
     ]
    }
   ],
   "source": [
    "print(\"Correlation between LLM-as-a-judge and the human raters:\")\n",
    "print(\n",
    "    f\"{examples['llm_judge_score'].corr(examples['human_score'], method='pearson'):.3f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这已经不错了，考虑到两个随机、独立变量之间的皮尔逊相关系数会是 0！\n",
    "\n",
    "但我们很容易做得更好。🔝"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 改进 LLM 评判者\n",
    "\n",
    "正如 [Aparna Dhinakaran](https://twitter.com/aparnadhinak/status/1748368364395721128) 所说的，LLM 在评估连续范围的输出方面表现不佳。\n",
    "[这篇文章](https://www.databricks.com/blog/LLM-auto-eval-best-practices-RAG)为我们提供了一些构建更好提示的最佳实践：\n",
    "- ⏳ **增加思考时间**，在最终答案前添加一个`评估`字段。\n",
    "- 🔢 **使用较小的整数刻度**，比如 1-4 或 1-5，而不是我们之前使用的大范围浮点刻度。\n",
    "- 👩‍🏫 **提供一个指导性的刻度**。\n",
    "- 我们甚至添加了一个激励 LLM 的“胡萝卜”(这里指给它一点额外的激励，就像给人一个奖励一样。)！"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMPROVED_JUDGE_PROMPT = \"\"\"\n",
    "You will be given a user_question and system_answer couple.\n",
    "Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.\n",
    "Give your answer on a scale of 1 to 4, where 1 means that the system_answer is not helpful at all, and 4 means that the system_answer completely and helpfully addresses the user_question.\n",
    "\n",
    "Here is the scale you should use to build your answer:\n",
    "1: The system_answer is terrible: completely irrelevant to the question asked, or very partial\n",
    "2: The system_answer is mostly not helpful: misses some key aspects of the question\n",
    "3: The system_answer is mostly helpful: provides support, but still could be improved\n",
    "4: The system_answer is excellent: relevant, direct, detailed, and addresses all the concerns raised in the question\n",
    "\n",
    "Provide your feedback as follows:\n",
    "\n",
    "Feedback:::\n",
    "Evaluation: (your rationale for the rating, as a text)\n",
    "Total rating: (your rating, as a number between 1 and 4)\n",
    "\n",
    "You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.\n",
    "\n",
    "Now here are the question and answer.\n",
    "\n",
    "Question: {question}\n",
    "Answer: {answer}\n",
    "\n",
    "Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your AI company.\n",
    "Feedback:::\n",
    "Evaluation: \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "examples[\"llm_judge_improved\"] = examples.progress_apply(\n",
    "    lambda x: llm_client.text_generation(\n",
    "        prompt=IMPROVED_JUDGE_PROMPT.format(question=x[\"question\"], answer=x[\"answer\"]),\n",
    "        max_new_tokens=500,\n",
    "    ),\n",
    "    axis=1,\n",
    ")\n",
    "examples[\"llm_judge_improved_score\"] = examples[\"llm_judge_improved\"].apply(\n",
    "    extract_judge_score\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation between LLM-as-a-judge and the human raters:\n",
      "0.843\n"
     ]
    }
   ],
   "source": [
    "print(\"Correlation between LLM-as-a-judge and the human raters:\")\n",
    "print(\n",
    "    f\"{examples['llm_judge_improved_score'].corr(examples['human_score'], method='pearson'):.3f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过对提示的少量调整，相关性**提高了近 30%**（其中几个百分点是因为我无耻地给了 LLM 一个小提示，我在此声明该提示不具有法律约束力）。\n",
    "\n",
    "相当令人印象深刻！👏\n",
    "\n",
    "让我们展示一些 LLM 评判者的错误来分析它们："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>human_score</th>\n",
       "      <th>explanation_1</th>\n",
       "      <th>llm_judge_improved_score</th>\n",
       "      <th>llm_judge_improved</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1976</th>\n",
       "      <td>What can I do to help people that are grieving?</td>\n",
       "      <td>Coping with Stress\\nTake care of yourself and your community\\nTaking care of yourself, your friends, and your family can help you cope with\\nstress. Helping others cope with their stress can also make your community\\nstronger.\\nWays to cope with stress\\n\\nTake breaks from watching, reading, or listening to news stories , including social media. Hearing about the pandemic repeatedly can be upsetting.\\nTake care of your body. \\nTake deep breaths, stretch, or meditate.\\nTry to eat healthy, well-balanced meals.\\nExercise regularly, get plenty of sleep.\\nAvoid alcohol and drugs.\\n\\n\\nMake time to unwind. Try to do some other activities you enjoy.\\nConnect with others. Talk with people you trust about your concerns and how you are feeling.\\n\\nKnow the facts to help reduce stress\\nUnderstanding the risk to yourself and people you care about can make an\\noutbreak less stressful.\\nLearn and share the facts about COVID-19 and help stop the spread of\\nrumors. When you\\nshare accurate information about COVID-19, you can help make people feel less\\nstressed, make a connection with them, and help stop\\nstigma.\\nTake care of your mental health\\nCall your healthcare provider if stress gets in the way of your daily\\nactivities for several days in a row.\\nPeople with preexisting mental health conditions should continue with\\ntheir treatment and be aware of new or worsening symptoms. Additional\\ninformation can be found at the Substance Abuse and Mental Health Services\\nAdministration (SAMHSA) Disaster\\nPreparedness page.\\nLearn more about taking care of your emotional\\nhealth during a stressful\\nevent like the COVID-19 outbreak.</td>\n",
       "      <td>1</td>\n",
       "      <td>The question is about others which the reply did not answer.</td>\n",
       "      <td>2.0</td>\n",
       "      <td>The system_answer is mostly not helpful. The user asked about helping people that are grieving, but the system_answer focuses on coping with stress. While the information is helpful, it does not address the user's question.\\nTotal rating:  2\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is mostly helpful. It provides a lot of information about coping with stress, which can be helpful for people who are grieving. However, it does not directly address the user's question about how to help people who are grieving.\\nTotal rating:  3\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is excellent. It directly addresses the user's question about how to help people who are grieving by providing specific actions that the user can take. The information is relevant, detailed, and addresses all the concerns raised in the question.\\nTotal rating:  4\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is terrible. It does not address the user's question at all. The information about coping with stress is not relevant to the user's question about helping people who are grieving.\\nTotal rating:  1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2026</th>\n",
       "      <td>How should I know whether I need to isolate myself or go into quarantine?</td>\n",
       "      <td>FAQs for Correctional and Detention Facilities\\nStaff at Correctional and Detention Facilities\\nWhat does it mean to be in quarantine?\\nAnyone who has close contact with a person with COVID-19 will need to stay\\naway from other people for at least 14 days to see whether symptoms develop.\\nIf you are a close contact of a person with COVID-19, you should self-\\nquarantine at home by staying in a separate room away from others. Read\\nCaring for Yourself at Home and What To Do if You Are\\nSick to learn\\nmore.</td>\n",
       "      <td>3</td>\n",
       "      <td>Answer is relevant to the question but is vague due to providing links for further reading. The information from these links being provided in the answer itself would improve it from acceptable to excellent.</td>\n",
       "      <td>2.0</td>\n",
       "      <td>The system_answer is mostly not helpful. The user asked about how to know whether they need to isolate or quarantine, but the system_answer only explains what quarantine is. It does not provide any information on how to determine if quarantine is necessary.\\nTotal rating:  2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5375</th>\n",
       "      <td>What symptoms are associated with Covid-19?</td>\n",
       "      <td>Q&amp;A: Older people and COVID-19\\nWhat is COVID-19?\\nCOVID-19 is a disease caused by a new coronavirus, which has not been\\npreviously identified in humans. In most cases, COVID-19 causes mild symptoms\\nincluding dry cough, tiredness and fever, though fever may not be a symptom\\nfor some older people. Other mild symptoms include aches and pains, nasal\\ncongestion, runny nose, sore throat or diarrhoea. Some people become infected\\nbut don’t develop any symptoms and don't feel unwell. Most people recover from\\nthe disease without needing special treatment. Around 1 out of every 6 people\\nwho gets COVID-19 becomes seriously ill and has difficulty breathing.</td>\n",
       "      <td>4</td>\n",
       "      <td>This answer has a list of symptoms in it.</td>\n",
       "      <td>3.0</td>\n",
       "      <td>The system_answer is mostly helpful: provides support, but still could be improved. The answer does provide a list of symptoms associated with Covid-19, but it also includes a lot of information that is not directly related to the question.\\nTotal rating: 3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                       question  \\\n",
       "1976                            What can I do to help people that are grieving?   \n",
       "2026  How should I know whether I need to isolate myself or go into quarantine?   \n",
       "5375                                What symptoms are associated with Covid-19?   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      answer  \\\n",
       "1976  Coping with Stress\\nTake care of yourself and your community\\nTaking care of yourself, your friends, and your family can help you cope with\\nstress. Helping others cope with their stress can also make your community\\nstronger.\\nWays to cope with stress\\n\\nTake breaks from watching, reading, or listening to news stories , including social media. Hearing about the pandemic repeatedly can be upsetting.\\nTake care of your body. \\nTake deep breaths, stretch, or meditate.\\nTry to eat healthy, well-balanced meals.\\nExercise regularly, get plenty of sleep.\\nAvoid alcohol and drugs.\\n\\n\\nMake time to unwind. Try to do some other activities you enjoy.\\nConnect with others. Talk with people you trust about your concerns and how you are feeling.\\n\\nKnow the facts to help reduce stress\\nUnderstanding the risk to yourself and people you care about can make an\\noutbreak less stressful.\\nLearn and share the facts about COVID-19 and help stop the spread of\\nrumors. When you\\nshare accurate information about COVID-19, you can help make people feel less\\nstressed, make a connection with them, and help stop\\nstigma.\\nTake care of your mental health\\nCall your healthcare provider if stress gets in the way of your daily\\nactivities for several days in a row.\\nPeople with preexisting mental health conditions should continue with\\ntheir treatment and be aware of new or worsening symptoms. Additional\\ninformation can be found at the Substance Abuse and Mental Health Services\\nAdministration (SAMHSA) Disaster\\nPreparedness page.\\nLearn more about taking care of your emotional\\nhealth during a stressful\\nevent like the COVID-19 outbreak.   \n",
       "2026                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          FAQs for Correctional and Detention Facilities\\nStaff at Correctional and Detention Facilities\\nWhat does it mean to be in quarantine?\\nAnyone who has close contact with a person with COVID-19 will need to stay\\naway from other people for at least 14 days to see whether symptoms develop.\\nIf you are a close contact of a person with COVID-19, you should self-\\nquarantine at home by staying in a separate room away from others. Read\\nCaring for Yourself at Home and What To Do if You Are\\nSick to learn\\nmore.   \n",
       "5375                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    Q&A: Older people and COVID-19\\nWhat is COVID-19?\\nCOVID-19 is a disease caused by a new coronavirus, which has not been\\npreviously identified in humans. In most cases, COVID-19 causes mild symptoms\\nincluding dry cough, tiredness and fever, though fever may not be a symptom\\nfor some older people. Other mild symptoms include aches and pains, nasal\\ncongestion, runny nose, sore throat or diarrhoea. Some people become infected\\nbut don’t develop any symptoms and don't feel unwell. Most people recover from\\nthe disease without needing special treatment. Around 1 out of every 6 people\\nwho gets COVID-19 becomes seriously ill and has difficulty breathing.   \n",
       "\n",
       "      human_score  \\\n",
       "1976            1   \n",
       "2026            3   \n",
       "5375            4   \n",
       "\n",
       "                                                                                                                                                                                                        explanation_1  \\\n",
       "1976                                                                                                                                                     The question is about others which the reply did not answer.   \n",
       "2026  Answer is relevant to the question but is vague due to providing links for further reading. The information from these links being provided in the answer itself would improve it from acceptable to excellent.   \n",
       "5375                                                                                                                                                                        This answer has a list of symptoms in it.   \n",
       "\n",
       "      llm_judge_improved_score  \\\n",
       "1976                       2.0   \n",
       "2026                       2.0   \n",
       "5375                       3.0   \n",
       "\n",
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              llm_judge_improved  \n",
       "1976   The system_answer is mostly not helpful. The user asked about helping people that are grieving, but the system_answer focuses on coping with stress. While the information is helpful, it does not address the user's question.\\nTotal rating:  2\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is mostly helpful. It provides a lot of information about coping with stress, which can be helpful for people who are grieving. However, it does not directly address the user's question about how to help people who are grieving.\\nTotal rating:  3\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is excellent. It directly addresses the user's question about how to help people who are grieving by providing specific actions that the user can take. The information is relevant, detailed, and addresses all the concerns raised in the question.\\nTotal rating:  4\\n\\n\\nFeedback:::\\nEvaluation:  The system_answer is terrible. It does not address the user's question at all. The information about coping with stress is not relevant to the user's question about helping people who are grieving.\\nTotal rating:  1  \n",
       "2026                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         The system_answer is mostly not helpful. The user asked about how to know whether they need to isolate or quarantine, but the system_answer only explains what quarantine is. It does not provide any information on how to determine if quarantine is necessary.\\nTotal rating:  2  \n",
       "5375                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           The system_answer is mostly helpful: provides support, but still could be improved. The answer does provide a list of symptoms associated with Covid-19, but it also includes a lot of information that is not directly related to the question.\\nTotal rating: 3  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "errors = pd.concat(\n",
    "    [\n",
    "        examples.loc[\n",
    "            examples[\"llm_judge_improved_score\"] > examples[\"human_score\"]\n",
    "        ].head(1),\n",
    "        examples.loc[\n",
    "            examples[\"llm_judge_improved_score\"] < examples[\"human_score\"]\n",
    "        ].head(2),\n",
    "    ]\n",
    ")\n",
    "\n",
    "display(\n",
    "    errors[\n",
    "        [\n",
    "            \"question\",\n",
    "            \"answer\",\n",
    "            \"human_score\",\n",
    "            \"explanation_1\",\n",
    "            \"llm_judge_improved_score\",\n",
    "            \"llm_judge_improved\",\n",
    "        ]\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们的 LLM 评判者的不一致之处很微小：总体来看，我们的系统似乎已经达到了不错的性能水平！\n",
    "\n",
    "## 4. 我们如何进一步提高 LLM 评判者的水平？\n",
    "\n",
    "🎯 **你永远达不到 100%：** 首先，我们的人类基准肯定包含一些噪音，所以即使有完美的 LLM 评判者，一致性和相关性也不可能达到 100%。\n",
    "\n",
    "🧭 **提供参考信息：** 如果你每个问题都有一个参考答案，你绝对应该在 LLM 评判者的提示中提供这些信息，以获得更好的结果！\n",
    "\n",
    "▶️ **提供少量示例：** 在提示中添加一些问题和基准评估的少量示例可以改善结果。 _(我这里尝试了，但在这个案例中并没有改善结果，所以我省略了，但这可能对你的数据集有效！)_\n",
    "\n",
    "➕ **累加刻度：** 当评判可以分解为原子性标准时，使用累加刻度可以进一步改善结果：如下所示 👇\n",
    "\n",
    "```python\n",
    "ADDITIVE_PROMPT = \"\"\"\n",
    "(...)\n",
    "- Award 1 point if the answer is related to the question.\n",
    "- Give 1 additional point if the answer is clear and precise.\n",
    "- Provide 1 further point if the answer is true.\n",
    "- One final point should be awarded if the answer provides additional resources to support the user.\n",
    "...\n",
    "\"\"\"\n",
    "```\n",
    "\n",
    "## 总结\n",
    "今天的内容就到这里，恭喜你跟到这里！🥳\n",
    "\n",
    "我必须得走了，一群奇奇怪怪的人正在敲我的门，声称他们是代表 Mixtral 来收取 H100s 的。🤔"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "cookbook",
   "language": "python",
   "name": "cookbook"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
